The combination of deep learning and Monte Carlo Tree Search (MCTS) has shown\nto be effective in various domains, such as board and video games. AlphaGo\nrepresented a significant step forward in our ability to learn complex board\ngames, and it was rapidly followed by significant advances, such as AlphaGo\nZero and AlphaZero. Recently, MuZero demonstrated that it is possible to master\nboth Atari games and board games by directly learning a model of the\nenvironment, which is then used with MCTS to decide what move to play in each\nposition. During tree search, the algorithm simulates games by exploring\nseveral possible moves and then picks the action that corresponds to the most\npromising trajectory. When training, limited use is made of these simulated\ngames since none of their trajectories are directly used as training examples.\nEven if we consider that not all trajectories from simulated games are useful,\nthere are thousands of potentially useful trajectories that are discarded.\nUsing information from these trajectories would provide more training data,\nmore quickly, leading to faster convergence and higher sample efficiency.\nRecent work introduced an off-policy value target for AlphaZero that uses data\nfrom simulated games. In this work, we propose a way to obtain off-policy\ntargets using data from simulated games in MuZero. We combine these off-policy\ntargets with the on-policy targets already used in MuZero in several ways, and\nstudy the impact of these targets and their combinations in three environments\nwith distinct characteristics. When used in the right combinations, our results\nshow that these targets can speed up the training process and lead to faster\nconvergence and higher rewards than the ones obtained by MuZero.\n