Trust Region Policy Optimization (TRPO) is a popular and empirically\nsuccessful policy search algorithm in reinforcement learning (RL). It\niteratively solved the surrogate problem which restricts consecutive policies\nto be close to each other. TRPO is an on-policy algorithm. On-policy methods\nbring many benefits, like the ability to gauge each resulting policy. However,\nthey typically discard all the knowledge about the policies which existed\nbefore. In this work, we use a replay buffer to borrow from the off-policy\nlearning setting to TRPO. Entropy regularization is usually used to improve\npolicy optimization in reinforcement learning. It is thought to aid exploration\nand generalization by encouraging more random policy choices. We add an Entropy\nregularization term to advantage over {\\pi}, accumulated over time steps, in\nTRPO. We call this update EnTRPO. Our experiments demonstrate EnTRPO achieves\nbetter performance for controlling a Cart-Pole system compared with the\noriginal TRPO\n