Recent successful deep reinforcement learning algorithms, such as Trust Region Policy Optimization or Proximal Policy Optimization, are fundamentally variations of conservative policy iteration. These algorithms iterate policy evaluation followed by a softened policy improvement step. As so, they are naturally on-policy. Here, we propose to combine (any kind of) soft greediness with Modified Policy Iteration. The proposed abstract framework applies repeatedly: (i) a partial policy evaluation step that allows off-policy learning and (ii) any softened greedy step. Our contribution can be seen as a new generic tool for the deep reinforcement learning toolbox.